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Abstract 

This  paper  describes  some  sequential  selection  procedures  for  selecting  the  normal  population 
having  the  largest  mean,  and  for  selecting  the  Bernoulli  population  having  the  largest  success  prob¬ 
ability,  with  emphasis  on  recent  developments.  Both  the  indifference-zone  and  subset  approaches 
are  discussed.  Some  results  for  the  exponential  family  including  a  decision-theoretic  approach  are 
also  described. 
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1.  INTRODUCTION 

Problems  of  statistical  inference  that  are  now  commonly  known  as  ranking  and  selec¬ 
tion  problems  gained  the  attention  of  statistical  researchers  in  the  early  1950's.  Early 
work  in  this  area  by  Bahadur  [1],  Bahadur  and  Robbins  [2],  Bechhofer  [4),  and  Gupta  [21] 
related  to  single-sample  procedures.  Interest  in  sequential  selection  procedures  arose  in 
the  early  days  and  has  steadily  continued  ever  since.  However,  it  was  a  decade  before  a 
substantial  amount  of  original  research  on  sequential  methods  for  ranking  and  selection 
problems  was  published  in  the  form  of  a  monograph  by  Bechhofer,  Kiefer  and  Sobel  [11] 
which  still  serves  well  as  a  constant  source  of  results  and  ideas. 

Two-stage  and  multi-stage  procedures,  in  general,  ran  be  viewed  as  closed  sequential 
procedures,  the  number  of  stages  needed  to  make  the  final  decision  being  bounded  above. 
Selection  procedures  involving  two  or  more  stages  arise  not  only  in  the  context  of  effi¬ 
ciency  compared  to  single-stage  procedures  but  also  arise  out  of  necessity  when  nuisance 
parameters  are  present  depending  on  the  requirements  set  on  a  procedure. 

Selection  procedures  have  been  studied  under  various  goals  such  as  selecting  the  best 
among  k  (>  2)  populations,  selecting  the  t  best  (1  <  t  <  k),  and  selecting  the  populations 
better  than  a  standard  or  control.  In  all  these  cases,  the  procedure  is  devised  to  select  a  sub¬ 
set  of  the  k  given  populations  which  is  of  either  a  fixed  size  or  a  random  size.  The  fixed-size 
subset  selection  in  the  classical  formulation  is  known  as  the  indifference-zone  112)  approach 
and  the  other  type  is  called  the  subset  selection  (SSl  approach.  More  will  be  said  about 
these  in  the  next  section. 
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Besides  the  monograph  of  Bechhofcr,  Kiefer  and  Sobel  [11]  devoted  entirely  to  se¬ 
quential  procedures,  a  few  other  books  on  ranking  and  selection  are:  Buringer,  Martin 
and  Schriever  [14],  Gibbons,  Olkin  and  Sobel  [20],  Gupta  and  Huang  [24],  and  Gupta  and 
Panchapakesan  [27].  The  last  book  [27]  mentioned  provides  a  comprehensive  survey  of  de¬ 
velopments  in  this  field  up  to  1978  with  an  extensive  bibliography.  Dudewicz  and  Koo  [16] 
have  given  a  categorized  bibliography.  Recently,  Gupta  and  Panchapakesan  [28]  have  sur¬ 
veyed  developments  in  the  subset  selection  theory  over  a  period  of  more  than  thirty  years 
with  emphasis  on  historical  perspectives.  A  nice  review  of  developments  in  the  multi-stage 
selection  theory  since  1979  is  given  by  Miesckc  [3S]. 

In  the  present  paper,  we  do  not  attempt  to  give  a  complete  account  of  sequential 
methods  in  ranking  and  selection.  Our  purpose  here  is  to  provide  a  basic  background,  give 
highlights  of  some  of  the  early  developments  and  their  impact  on  some  current  develop¬ 
ments. 

Section  2  gives  a  general  background  for  sequential  selection  procedures,  explaining 
the  basic  aspects  of  the  indifference-zone  ami  subset  approaches.  The  specific  procedures 
discussed  here  center  around  selecting  the  normal  population  having  the  largest  mean, 
and  selecting  the  Bernoulli  population  having  the  largest  success  probability.  These  are 
discussed  respectively  in  Sections  3  and  4  for  the  indifference-zone  approach,  and  in  Sec¬ 
tions  5  and  6  for  the  subset  selection  approach.  Section  7  deals  with  subset  selection  from 
exponential  family  distributions  and  a  decision-theoretic  approach  to  the  problem. 

2.  SOME  GENERAL  ASPECTS  OF  SEQUENTIAL  SELECTION  PROCEDURES 

Let  7Ti, . . .  ,7Tfc  be  k  given  populations.  From  each  t;,  a  sequence  of  independent 
observations  Xu ,  X&, ...  is  available  to  the  experimenter.  Let  -Y, >  have  a  density  with 
respect  to  (w.r.t.)  a  <7-finite  measure  on  R.  which  is  the  Lebesgue  measure  or  a  counting 
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measure.  The  parameters  0, ,  i  =  1, . . . ,  k,  are  assumed  to  he  unknown.  Let  0pj  <  . . .  < 
denote  the  ordered  0j.  No  prior  knowledge  is  assumed  regarding  the  true  pairing  of  the 
ordered  and  unordered  0j.  Our  goal  is  to  select  the  population  ir <  which  has  the  largest 
associated  0j  and  is  called  the  best  population.  In  ease  of  a  tie,  we  consider  one  such 

population  is  tagged  as  the  best.  Let  ft  =  {0  :  0  =  (0i _ ,0fch  0«  €  0,  i  =  l,...,Jfc} 

denote  the  parameter  space,  where  0  is  taken  to  be  some  interval  (finite  or  infinite)  on 
the  real  line.  Let  D(6,,0j)  >  0  be  an  appropriately  defined  distance  measure  between  the 
populations  iti  and  w}.  For  6*  >  0,  define 

ft(^)={«|D(%._.).%.])>A*}-  (1) 

In  the  case  of  location  parameters  0, ,  for  example,  a  natural  choice  is  Z?(0[fc_i),0[t])  = 
0|k]  - 

Under  the  IZ  approach  of  Bechhofer  {4),  a  valid  procedure  R  selects  one  of  the  k 
populations  as  the  best  with  a  guarantee  that 

Pg(CS\R)  >  P*  whenever  0  E  Q(t>* )  (2) 

where  Pq(CS\R)  denotes  the  probability  of  a  correct  selection  (PCS)  using  the  rule  R 
under  the  parametric  configuration  0.  The  minimum  probability  level  P*(£  <  P*  <  1) 
and  6*  are  specified  in  advance*  by  the  experimenter.  The  complement  of  ft(0*)  in  (1)  w.r.t. 
ft  is  called  the  indifference-zone  since  we  have  no  PCS  requirement  for  0  in  this  part.  The 
part  ft(0*)  is  known  as  the  preference- zone. 

In  the  subset  selection  ( SS )  approach  of  Gupta  [21.  22J,  a  valid  procedure  R  selects  a 
random-sized  subset  of  the  given  populations  with  a  guarantee  that 

P#(CS\R)  >  P*  for  all  0  E  ft  (3) 
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where  a  correct  selection  ( CS )  occurs  if  the  best  population  is  included  in  the  selected 
subset.  We  note  that  there  is  no  indifference- zone  in  the  SS  approach. 

The  probability  requirements  { 2 )  and  ( 3 )  are  usua lly  referred  to  as  the  basic  probability 
requirements  or  P* -requirements  or  P*-conditions  of  the  respective  formulations.  In  either 
of  these  classical  approaches,  one  proposes  a  ‘’reasonable”  procedure  which  involves  some 
quantities  to  be  defined  so  that  the  P* -requirement  is  met.  This  involves  the  all-important 
first  step  of  finding  the  least  favorable  configuration  (LFC)  of  9  (in  fl(6*)  or  Q,  depending 
on  the  approach)  for  which  the  infimum  of  P(CS\R)  over  the  appropriate  space  takes 
place.  The  necessary  quantities  involved  in  the  rule  R  are  then  determined  such  that  this 
infimum  is  at  least  P*.  Since  the  procedure  is  proposed  ad  hoc,  one  would  then  study 
its  properties,  evaluate  its  performance  according  to  suitable  criteria,  and  compare  the 
performance  with  that  of  any  known  alternative  procedure's. 

A  selection  procedure,  though  not  always  explicitly  so  stated,  typically  consists  of 
three  parts:  (1)  a  sampling  rule.  (2)  a  stopping  rule,  and  (3)  a  terminal  decision  rule. 
Procedures  are  usually  categorized  according  to  tin'  types  of  rules  employed  in  the  above 
three  parts.  The  terminal  decision  identifies  a  procedure  as  a  fixed  size  or  a  random  size 
subset  selection  procedure.  A  sequential  procedure  is  said  to  be  closed  or  open  according 
as  the  number  of  observations  that  can  be  drawn  from  each  population  is  a  bounded 
or  an  unbounded  random  variable.  When  the  number  of  stage’s  involved  in  a  sequential 
procedure  is  a  bounded  random  variable  it  is  said  to  be  n  truncated  procedure.  A  sequential 
procedure  with  elimination  may  eliminate  one  or  more  populations  (which  appear  to  be 
inferior)  before  reaching  the  final  stage  at  which  the  terminal  decision  is  made.  Typically, 
further  sampling  from  eliminated  populations  is  discontinued  although  this  is  not  the  case 
with  some  procedures  studied  in  the  literature. 
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Sampling  may  be  done  one-at-a-time  or  vector-at-a-time.  The  former  is  an  adaptive 
sampling  in  which  the  population  to  be  sampled  from  next  depends  on  the  data  accumu¬ 
lated  until  then.  Plav-the-winner  sampling  rule  of  Robbins  [42]  in  the  case  of  Bernoulli 
populations  is  an  instance  of  this  case.  In  the  vector-at-a-time  sampling,  a  vector  of  ob¬ 
servations  (one  from  each)  is  taken  from  the  uou-eliminated  populations. 

3.  SELECTION  FROM  NORMAL  POPULATIONS:  IZ  APPROACH 
Let  ttj  , . . . ,  TTjt  be  k  normal  populations  with  unknown  means  0i , . . . ,  0*,  respectively, 
and  a  common  variance  <r2 .  For  defining  the  preference- zone  in  (1),  we  take  D(0[* - i ],<?[*] )  = 
#[*]  -  0[*-i];  thus 

n«*)  =  {$i%]  -%-»]>  6*  >o}. 

Our  goal  is  to  select  the  population  associated  with  0n.j  and  any  valid  rule  should  satisfy 
the  P* -requirement  (2).  We  will  discuss  the  known  ol  case  first. 

3.1  Case  A:  Known  a2.  Let  X,j,  j  =  1,2, . . .  be  a  sequence  of  independent  observations 
from  Wj,  i  —  1  ,...,fc.  Unless  stated  otherwise,  the  observations  are  taken  vector-at-a- 

m 

time.  Let  F,m  =  Xij,  i  =  1 and  let  lpjm  <  ...  <  V(t]m  denote  the  ordered 

Stein’s  Procedure.  Rjviz.s-  Stein  [47],  using  a  slightly  more  general  model  than  ours, 
proposed  an  open  sequential  procedure  with  elimination  which  is  a  straightforward  appli¬ 
cation  of  a  lemma  of  Wald.  At  stage  m  (in  =  1,2 _ ),  let 

m  _ 

A,  =  £  [A',  j  -X  j-  t>'(tj  -  1  )/tj}.  i  =  1 . k, 

>=  i 

where  X }  is  the  average  of  the  observations  at  stage j,  and  tj  is  the  number  of  populations 
sampled  from  at  this  stage.  Stein’s  procedure  is  described  below: 

Procedure  R  vi7.c:  At  stage  in  (in  =  1,2, . . .),  eliminate  all  populations  Xi  for  which 
Ai  <  {a2 /8*)in(  1  —  P*)  and  proceed  to  stage  (in  +  1)  to  take  an  additional  observation 
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from  each  remaining  population.  Stop  the  experiment  at  any  stage  m,  if  there  is  at  most 
one  i  for  which  A,  >  (a2 / 6*  )£n(l  —  P*).  If  there  is  exactly  one  such  i  at  termination,  then 
select  that  7r,  as  the  best;  otherwise  (i.e.  no  such  A; ),  select  ir,  corresponding  to  the  largest 
Ai  at  termination. 

Bechhofer-Kiefer-Sobel  Procedure.  Ej viz-.bks-  In  their  monograph,  Bechhofer,  Kiefer 
and  Sobel  [11]  considered  selection  from  populations  belonging  to  a  Koopman-Darmois 
family.  Their  procedure  is  an  onen  sequential  one  with  no  elimination.  This  procedure 
is  specialized  by  them  [11,  pp.  2G4-265]  to  tin*  normal  case  at  hand.  For  each  m  (m  = 
1,2,...),  let  Wm  =  E  exp{-<5*(}'u.]m  -  ![.),,.  )/<**} - 

■=i 

Procedure  Rv jz-.bks'  Stop  sampling  when  m  =  N,  the  first  positive  integer  for  which 
Wm  <  (1  —  P*)/P*;  select  the  population  corresponding  to  the  largest  Yi\. 

A  drawback  of  the  above  procedure,  as  noted  by  Bechhofer  and  Goldsman  [7],  is  that  if 
is  small,  then  N  (the  stopping  time)  can  be  large  with  a  considerable  probability. 
Further,  the  variance  of  N  can  be  large.  To  overcome  these  undesirable  effects,  Bechhofer 
and  Goldsman  [7]  proposed  a  truncated  version  (described  below)  of  the  above  procedure. 

Bechhofer-Goldsman  Procedure.  Rsiz.bg-  This  procedure  modifies  the  stopping  rule 
of  Rniz-bks  as  follows:  Stop  sampling  when,  for  the  first  time,  either  Wm  <  (1  —  P*)/P* 
or  m  as  no,  whichever  occurs  first.  Here  no  =  n«(fc,  6 *,  P* )  is  predetermined  as  the  smallest 
positive  integer  which  guarantees  the  P* -requirement  (2).  The  terminal  decision  rule  is: 
Select  the  population  corresponding  to  the  largest  where  N  is  now  the  bounded 
stopping  time. 

Bechhofer  and  Goldsrr.nn  [7]  have  tabulated  the  /«»  values  for  k  =  2(1)5,  6*  =  0.2(0.2) 
0.8,  and  P*  =  0.75,0.90,0.95,0.99. 

Another  well-known  procedure  in  the  literature  is  that  of  Paulson  [39],  who  was  the 
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first  to  consider  a  closed  procedure  with  elimination,  a  feature  to  be  characterized  by  some 
later  authors  as  Paulson- tvne.  Paulson,  in  fact,  considered  a  class  of  procedures  indexed  by 
A  €  (0, 6 *),  using  triangular  stopping  regions.  Let  a\  —  { cr2/(6 *  —  A)}£n{(fc  — 1)/(1  —  P* )} 
and  let  W\  denote  the  largest  integer  less  than  «a/A. 

Paulson  Procedure.  R Iniz-.p-  At  the  beginning  of  stage  m  (m  =  1, . . . ,  Wa),  take  one 
observation  from  each  population  not  eliminated  thus  far.  Now  eliminate  all  populations 
iti  for  which 

Yi,„  <  max  Yrm  -  +  »iA 

r 


where  the  maximum  is  over  all  populations  jry  that  remain  at  the  beginning  of  stage  m. 
If  all  but  one  population  are  eliminated,  then  stop  sampling  and  select  this  one  remaining 
population;  otherwise,  proceed  to  stage  (rn  +  1).  If  two  or  more  populations  remain 
after  stage  Ik  a,  then  take  an  additional  observation  from  each  one  of  them  and  select  the 
population  7f,  corresponding  to  the  largest  I'( u\+i )• 

Although  Rniz.p  guarantees  the  /^-requirement,  the  optimum  value  of  A  in  (0, 6*) 
was  not  settled  by  Paulson.  However,  based  on  his  calculations,  he  recommended  the 
choice  of  A  =  6* / 4.  Bechhofer  and  Goldsman  [10]  prefer  A  =  6* /2  because  this  minimizes 
IVa  -f  1,  the  maximum  possible  total  number  of  stages  to  termination,  for  any  given  set  of 
k ,  ()*,  and  P*. 

Improvements  in  Paulson  s  Procedure.  Fabian  [19]  improved  Paulson’s  procedure  by 
obtaining  better  lower  bound  on  the  PCS.  Considering  the  choices  of  A  =  6* / 2  and  A  = 
S*/4,  Fabian’s  improvement  is  achieved  by  replacing  r  =  (k  -  1)/(1  -  P*)  by  c/2  for 
A  =  6*/2  and  by  1/q  for  A  =  d*/4,  where  </  satisfies  (</  —  )c  =  1.  Recently,  Hartman 
[30]  improved  upon  Fabian’s  results  by  replacing  the  reciprocal  of  c  by  1  -  (P*)1/(k“1), 
Some  Comparison  Results.  Bechhofer  and  Goldsman  [10]  have  done  Monte  Carlo 
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studies  to  compare  the  performances  of  Rnix-.dk s,  R\ iz-.hg,Rniz-.ph  (i-e-  Rniz-.p  with 
Hartman  modification),  the  two-stage  procedure  of  Tamhane  and  Bechhofer  [49,  50]  and 
the  single-stage  procedure  of  Bechhofer  [4].  Let  us  call  the  last  two  procedures  Rniz-.tb 
and  Rniz.Bi  respectively.  The  performances  of  these  procedures  were  studied  by  Bechhofer 
and  Goldsman  [10]  in  terms  of  achieved  PCS.  E(  AT),  and  E(T ),  where  N  and  T  are  the 
total  number  of  stages  needed  to  terminate  and  the  total  number  of  observations  taken  up 
to  termination.  Their  results  indicate  that  Rniz-.uc  does  well  in  terms  of  E(N)  except 
when  the  are  all  very  close  to  each  other  and  P*  is  high,  in  which  case  Rniz-.PH  with 
A  =  6* / 2  is  recommended.  When  k  >  5,  they  recommend  Rniz-.pu  with  A  =  6*/ 2  for  the 
equal  means  (EM)  configuration  and  A  =  £*/4  otherwise.  For  reasonably  high  P*  with 
E(T)  as  the  criterion,  Rniz.ph  seems  preferable  with  choices  of  A  as  indicated  above. 

Kao-Lai  Procedure.  ILv  iz.KL-  A  class  of  truncated  procedures  with  elimination  was 
proposed  by  Kao  and  Lai  [34]  employing  confidence  sequences  for  the  (k  —  1)  differ¬ 
ences  0[*]  —  0,  (i  ^  [fc]).  Taking  E(T)  as  a  meiisure  of  efficiency,  it  has  been  shown 
by  Kao  and  Lai  [34]  that  asymptotically  (P*  — ►  1)  their  procedure  is  more  efficient  than 
Rn i z.BKSi  Rnjz-.Pi  and  Rniz.d  except  when  6  is  in  the  least  favorable  (slippage)  config¬ 
uration  or  in  the  EM -configuration;  in  these  configurations,  their  procedure  is  at  least  as 
efficient  as  the  others. 

A  Generalized  Goal.  Fabian  [18]  considered  a  generalized  goal  for  ranking  populations. 
For  our  problem  of  selecting  the  best  population,  this  corresponds  to  6* -correct  selection 
( 6*—CS )  which  means  selecting  any  n,  for  which  0,  >  0[*]  -  **•  Such  a  ni  is  called  a 
good  population.  For  6  €  )  in  the  IZ  approach,  the  best  population  is  the  only  good 

population.  Fabian  [18]  has  shown  that,  for  the  single-stage  procedure  of  Bechhofer  [4],  a 
stronger  claim  can  be  made,  namely,  that  P(6*  —CS\R)  >  P *  for  all  0  £  Q. 
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Kao  and  Lai  [34]  have  given  a  sequential  procedure  (by  slightly  modifying  the  elim¬ 
ination  rule  of  Rniz-.kl)  which  guarantees  a  minimum  probability  P *  of  a  6*—CS.  As 
pointed  out  by  Edwards  [17],  this  is  done  at  the  expense  of  considerably  slower  elimination 
of  inferior  populations.  Edwards  [17]  gave  a  slightly  different  but  more  general  procedure, 
which  he  called  an  extended-Panlson  sampling  plan.  His  procedure  guarantees  a  minimum 
probability  of  a  6*—CS  while  keeping  asymptotic  (P*  — »  1)  sample  size  properties  same 
as  those  of  the  IZ  procedures  Rniz-.bkSi  Rn  iz-.p-,  and  Rniz-.b- 

Other  Developments.  Hoel  [31]  has  discussed  a  method  of  constructing  Paulson-type 
procedures  based  on  log-likelihood  ratios,  which  can  be  applied  to  the  normal  means  prob¬ 
lem.  For  appropriate  choices  of  the  index  defining  a  family  of  procedures,  Hoel’s  procedure 
is  precisely  Rniz.p- 

Recently,  Bechhofer  and  Goldsman  [8]  have  considered  selection  of  normal  population 
with  the  largest  mean  when  the  underlying  model  is  a  two-factor  experiment  with  no 
interaction.  Their  procedure  is  a  natural  adaptation  of  Rniz.bks ■  In  a  later  paper  [9], 
they  studied  a  truncated  version  of  this  adapted  procedure  and  carried  out  Monte  Carlo 
studies  of  performances  of  these  procedures  and  a  single-stage  procedure  of  Bechhofer  [4]. 

3.2  Case  B:  Unknown  n1 .  When  a 2  is  unknown,  there  does  not  exist  a  single-stage  pro¬ 
cedure  that  can  guarantee  the  P* -requirement  under  the  IZ  formulation.  This  is  because 
the  necessary  sample  size  cannot  be  determiiH'd  without  the  knowledge  of  a2 .  Bechhofer, 
Dunnett  and  Sobel  [5]  proposed  a  two-stage  procedure  where  the  first  stage  samples  are 
used  to  provide  an  estimate  of  a2\  the  additional  second-stage  sample  size,  if  necessary, 
can  be  determined  accordingly.  Paulson  [39].  and  Kao  and  Lai  [34]  have  given  procedures 
by  modifying  their  earlier  procedures  for  the  case  of  known  a2 .  These  involve  first  taking 
m  (>  2)  observations  from  each  population  and  then  procf*eding  sequentially  by  taking  one 


observation  from  each  naneliminated  population.  Robbins,  Sobel  and  Starr  [43]  proposed 
a  procedure  for  which  the  P* -requirement  is  asymptotically  (6*  — »  0)  satisfied.  Details  of 
these  procedures  will  not  be  discussed  here.  These  procedures,  except  that  of  Kao  and  Lai 
[34],  have  been  discussed  in  Gupta  and  Panchapakesan  [27,  Chapter  6]. 


SELECTION  FROM  BERNOULLI  POPULATIONS:  IZ 


Let  be  it  Bernoulli  populations  with  associated  success  probabilities  p\ , . . ., 

Pk ,  respectively.  Consider  the  preference- zone  tt**  =  {p  :  p  =  (pi, . . . , pk ),  P[t]  —  P[t— i]  > 
S* } ,  where  pm  <  ...  <  pjfcj  arc  the  ordered  p, .  and  0  <  b*  <  1  is  specified  in  advance. 
For  selecting  the  population  associated  with  pj*.j  (the  best  population),  Sobel  and  Huyett 
[46]  studied  a  single-stage  procedure  bast'd  on  a  sample  of  size  n  from  each  population. 
This  procedure  is  Rdiz.sh ■  Select  the  population  corresponding  to  the  largest  number  of 
observed  successes. 


For  this  problem,  Paulson  [40,  41]  proposed  truncated  sequential  procedures  with 
elimination.  There  are  also  a  number  of  other  procedures  studied  by  several  authors;  these 
procedures  differ  in  their  sampling  and/or  stopping  rules.  A  detailed  discussion  of  these 
procedures  can  be  found  in  Gupta  and  Panchapakesan  [27,  Chapter  4]. 


Recently,  Bechhofcr  and  Kulkarni  [12]  proposed  a  dosed  sequential  procedure.  Their 
sampling  rule  involves  taking  at  each  stage  one  observation  from  a  population  to  be  deter¬ 
mined  by  the  accumulated  data  up  to  that  stage:  in  other  words,  it  is  a  one-at-a-time  adap¬ 
tive  sampling.  Also,  a  maximum  n  is  set  for  the  number  of  observations  that  can  be  drawn 


from  any  population. 


Let  riim  and  Zim  denote,  respectively,  the  total  number  of  observations  taken  from 
ni  and  the  number  of  successes  among  them  through  stage  m,  ?  —  l,...,fc  and  m  = 
0,1,..., Am.  Stage  0  (i.c.  no  observation  is  yet  taken)  is  introduced  for  convenience  in 
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describing  the  procedure  Rbiz.uk  of  Bechhofcr  and  Kulkarni  [12],  which  is  as  follows: 

a.  At  stage  m  (0  <  m  <  kn  —  1),  take  the  next  observation  from  the  population  which 
has  the  smallest  number  of  failures  among  all  7Tj  for  which  n*m  <  n.  In  case  of  a  tie 
among  such  ir^’s,  take  the  next  observation  from  the  one  which  has  the  largest  number 
of  successes.  In  case  of  a  further  tie,  select  one  of  this  further  tied  set  at  random  and 
draw  the  next  observation  from  that  population. 

b.  Stop  sampling  at  the  first  stage  m  at  which  there  exists  at  least  one  rr,  satisfying 

Zim  >  Z ji„  +  »  -  iij,„  for  all  j  ±  i.  (4) 

c.  Select  as  the  best  one  at  random  from  those  7r,’s  which  satisfy  (4)  at  termination. 
Bechhofer  and  Kulkarni  [12]  have  shown  that  the  PCS  for  Rbiz-.BK  equals  that  of 

Rbiz-.sh  uniformly  in  j>  for  k  >  2.  Several  optimal  properties  of  Rbiz-.bk  have  been 

established  by  Kulkarni  and  Jennison  [3G].  Exact  numerical  results  are  given  by  Bechhofer 

and  Kulkarni  [13]  for  performance  characteristics  such  as  the  distributions  of  A^j,  the 

number  of  observations  taken  at  truncation  from  7r,  associated  with  pj,j ,  and  of  the  total 
k  . 

number  N  —  £  Af(i)  at  truncation.  Because  of  the  nature  of  time-consuming  recursive 
12=1 

formulae,  their  numerical  results  are  limited  to  the  cases  of  (k, n)  =  (2,20)  and  (3,7)  for 
the  distributions  of  iV(l)  and  Ar,  and  are  limited  to  (k  =  2,  n  <  100)  and  (k  =  3,  n  <  40) 
for  J E{iV(,)}  and  E{N).  The  scope  of  these  studies  is  extended  to  k  =  4  and  5  by  Bechhofer 
and  Frisardi  [0]  employing  Monte  Carlo  simulation. 

The  sampling  rule  of  the  Bechhofer-Kulkarni  procedure  is  not  a  play-the-winner  rule 
(see  Bechhofer  and  Kulkarni  [13]);  it  is  referred  to  as  the  least  failures  rule  by  Kelly  [35] 
who  proposed  it  for  a  Bernoulli  multi-armed  bandit  problem. 

The  idea  behind  the  stopping  rule  of  the  Bwhliofer-Kulkarni  procedure  is  that  the 
sampling  can  be  curtailed  as  soon  as  there  exists  one  or  more  populations  which  have 
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at  least  as  many  successes  as  the  maximum  possible  number  of  successes  from  any  of  the 
other  populations  even  if  all  the  n  observations  were  taken  from  them.  This  criterion 
as  given  in  (4)  is  referred  to  as  strong  curtailment  by  Jennison  [32]  who  also  consid¬ 
ered  weak  curtailment  given  by  (4)  with  >  replaced  by  >.  With  either  curtailment,  the 
Bechhofer-Kulkarni  procedure  achieves  the  same  PCS  as  does  the  Sobel-Huyett  single-stage 
procedure  uniformly  in  p  =  (pi, ...  ,pt).  As  noted  by  Jennison  [32],  strong  curtailment  is 
preferable  to  weak  curtailment  since  the  former  yields  a  sample  size  no  larger  than  that 
yielded  by  the  latter. 

Jennison  and  Kulkarni  [33]  have  considered  similar  procedures  for  the  goal  of  selecting 
the  3  (1  <  s  <  k  —  1)  best  of  k  Bernoulli  populations.  Recently,  David  and  Andrews  [15] 
have  proposed  procedures  with  strong  and  weak  curtailments  for  selecting  the  best  of  k 
objects  in  a  Round  Robin-type  paired  comparison  experiment.  They  have  shown  that  the 
probabilities  of  selecting  a  particular  object  arc'  the  same  tinder  both  curtailments  for  the 
Bradley-Terry  model,  but  are  not  so,  in  general. 

5,  SELECTION  from  NORMAL  POPULATIONS;  gg. APPROACH 

Let  be  k  normal  populations  where  n,  has  mean  6t  and  variance  of,  i  = 

1, . . . ,  k.  Before  discussing  sequential  procedure's  for  different  goals,  we  state  the  single- 
stage  procedure  of  Gupta  [21]  when  <Tj  =  . . .  =  fj(  =  rr2  ( known!.  His  procedure  is  based 
on  Xi,  i  =  1, . . .  ,n,  the  means  of  random  samples  of  size  n  from  the  k  populations  and  is 
given  below. 

Gupta’s  Single-stage  Procedure.  B j^ss-.o-  Select  n,  if  and  only  if 

Xi  >  max  X j  -  ^7=  (5) 

1  <j<k  3  yfn 

where  the  constant  D  =  D(k,P*)  is  the  smallest,  positive  number  for  which  the  P*- 
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requirement  (3)  is  satisfied.  This  constant  D  is  given  by 


OO 


(x  +  D)  <mx)  =  p* 


where  $  denotes  the  standard  normal  distribution  function. 

Let  pi  denote  the  probability  of  selecting  the  population  associated  with  0m,  i  = 
1 ,k.  Then  it  is  known  that  pi  <  pj  <  . . .  <  pt  (i.e.  the  procedure  is  monotone!,  where 


OO 

Pi=  J  .A  ${x  +  D  +  (0pj  -  )N/n  or-1 } 


d*(x). 


5.1  Barron-Gtmta  Procedure.  R_nss.bg-  This  procedure  is  devised  for  selecting  a  subset 
containing  the  best  (i.e.  one  having  the  largest  8, )  assuming  that  =  . . .  =  <r|  =  a2 
(known)  and  that  the  successive  differences  of  the  ordered  0<  are  known  (this  means  that 
the  pi  in  (7)  are  known).  Their  procedure  employs  vector-at-a-time  sampling.  As  before, 
let  Xi\ ,  Xa,. . .  be  a  sequence  of  observations  from  7r<.  At  stage  j ,  we  have  the  observations 


Xij,  i  =  1,..  .,k.  Define 


{1  if  >  max}', 

,  .  r 

0  otherwise 


-  Da 


i  = 


where  D  is  given  by  (6).  In  other  words,  Y,j  —  1  if  tt,  is  selected  by  Gupta's  rule  in  (5) 
based  on  stage  j  observations  ( n  =  1 )  and  Y,  j  =  0,  otherwise. 

m 

Now,  for  any  stage  m,  define  S„„  =  £  Yt,  so  that  5j„,  has  a  binomial  distribution 

i- 1 

B{m,pi)  with  parameters  m  and  p,  (given  by  (7)).  This  fact  is  used  by  Barron  and  Gupta 
[3]  in  constructing  their  procedure.  As  we  will  see,  this  procedure  continues  sampling  from 


mi  decision  is  made  w 


wii'i'i  I  El  »;o  i  w  »T:9 


Barron  and  Gupta  [3],  in  fact,  defined  a  class  of  procedures  based  on  a  pair  of  sequences 
of  real  numbers,  ({bm },  {rm } ),  satisfying  for  m  >  1  the  following  conditions:  (i)  bm  < 
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I 

l 


&m+ 1.  Cm  <  cro+i,  (ii)  6m  <  cm,  (iii)  lim  bm  =  oo,  and  (iv)  Pr{  n  [bm  <  Sim  <  cm]}  =  0 

m— »oo  ro=l 

for  all  t  =  1, . . . ,  k.  For  each  such  pair,  the  Barron-Gupta  rule  is  as  follows. 

Procedure R v gg. an:  At  stage  to  (to  =  1.2, . tag  each  untagged  population  i r*  for 
which  Sim  &  (bm,cm)\  tag  it  “rejected”  if  5„„  <  b,„  and  “accepted”  if  5,m  ^  cm.  Stop 
sampling  when  all  the  populations  are  tagged.  At  termination,  select  all  those  populations 
that  were  tagged  “accepted.” 

It  should  be  noted  that  once  a  population  it i  is  tagged,  it  remains  so  irrespective  of 
later  changes  in  Sjm.  Barron  and  Gupta  [3]  have  studied  in  detail  several  properties  of  this 
procedure  including  its  performance  compared  with  the  single-stage  procedure  Rnss-.g- 
5.2  Swanepoel-Geertsema  Procedure.  Rjvss:SY.-  This  procedure  is  devised  for  select¬ 
ing  a  subset  containing  the  population  with  the  largest  6,  assuming  that  the  a*  are 
unknown  and  possibly  unequal.  It  is  a  sequential  procedure  with  no  elimination  employ¬ 
ing  vector-at-a-time  sampling,  and  is  based  on  constructing  a  selection  sequence.  For  each 
n  >  1,  let  B„  be  a  subset  of  the  k  populations  defined  by  »  observations  from  each.  Any 
sequence  {Bn}  is  a  selection  seonence  if 


6  B„  for  all  n  >  1}  >  P* 


for  all  6  €  ft  where  7t(*)  denotes  the  best  population  and  0  <  P*  <  1  is  given. 

Swanepoel  and  Geertsema  [48]  construct  a  selection  sequence  {J3„}  where  B\  = 
{iri,...,tr*}  and 


Bn  —  { 7fr  :  Xr(n)  >  max  X(n)  -  *„rh{k,P*,n)} 


where  jTj(n)  is  the  mean  of  n  observations  from  7r,.  s„r  is  an  estimator  of  max  0i  ^T  i 
and  h  is  a  constant  depending  on  fc,  P*,  and  n.  The  stopping  time  N  is  defined  to  be  the 
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first  integer  n  >  1  such  that  |B„|  <  m,  where  |B„|  denotes  the  size  of  B„  and  m  is  an 
integer  chosen  in  advance  with  1  <  m  <  k  —  1.  At  termination,  we  select  the  subset  Bat. 

In  the  unknown  true  configuration  of  0,  let denote'  the  number  of  0,’s  equal  to  0[*j.  If 
s  <  m,  then  N  <  oo  a.s.,  |B\|  <  m,  and  Bn  includes  the  best  population  with  minimum 
probability  P*. 

5.3  Gupta-Liang  Procedure.  &nss-.gl-  Gupta  and  Huang  [23]  proposed  and  studied  two 
procedures  based  on  log-likelihood  ratios  which  can  be  applied  to  location  and  scale  pa¬ 
rameter  cases.  One  of  these  two  procedures  is  with  elimination.  Their  goal  is  to  select  all 
mildly  t  best  populations  (i.e.  those  ir^’s  for  which  0,  >  0[*.— *+i]  — 6 *  for  a  specified  6*  >  0, 
in  the  location  case). 

Recently,  Gupta  and  Liang  [25]  have  considered  a  similar  setup  (with  some  slightly 
modified  assumptions)  and  proposed  a  sequential  procedure  applicable  to  location  and 
scale  cases  but  with  a  modified  goal.  For  the  location  case  with  t  =  1,  the  Gupta-Huang 
goal  is  to  select  all  good  populations.  The  Gupta-Liang  goal  is  to  select  a  subset  which 
includes  the  best  population  and  at  the  same  time  excludes  all  that  are  not  good.  An 
event  of  selecting  a  subset  consistent  with  this  goal  is  denoted  by  CS(6*)  [Note  that 
CS(#*)  is  different  from  6*— CS]. 

For  the  normal  means  problem  with  a  common  known  variance  a2,  let  Xu, Xu,... 
be  a  sequence  of  independent  observations  from  tt,,  i  =  1, . . .  ,k.  For  m  >  1,  define 

m 

Yim  —  E  Xij.  Let  Sm  denote  the  set  of  contending  populations  at  the  beginning  of  stage 
j=! 

m  and  |5m|  denotes  the  size  of  S,„.  We  now  define  the  Gupta-Liang  procedure. 

Procedure  R nss.gl  -  Choose  a  in  the  interval  (0, 6*/ 2).  At  stage  m  (m  =  1, 2, . . .), 
take  one  observation  from  each  population  in  S,„.  Include  in  Sm+i  only  those  jt,’s  in  Sm 


15 


for  which 


j(Yrm  -  Yim )  -  ^  <  log  i-~.  for  all  *r  6  Sm,r  i ; 

and  eliminate  all  other  ifj’s  from  any  further  consideration.  Now,  label  as  good  only  those 
jr,  ’s  in  that  have  not  been  labeled  so  far  and  for  which 

'‘■r-  (Yim  -  Vim)  +  ~~~  ~  —  >  log  for  all  1 Tt  e  Sm+1 ,  t  £  *. 

Stop  sampling  if  either  |Sm+i|  =  1  or  5m+i  do<*s  not  contain  any  unlabeled  population, 
and  make  the  terminal  decision:  “Select  all  the  populations  in  Sm+j”;  otherwise,  go  to 
stage  m+1. 

It  should  be  noted  that  a  population  is  not  labeled  unless  and  until  it  qualifies  to  be 
called  good.  Once  so  labeled,  it  is  not  examined  for  labeling  again.  It  is  also  possible  that  a 
labeled  population  is  eliminated  at  a  later  stage.  The  populations  that  are  selected  are  the 
ones  which  have  been  found  to  be  good  at  some  stage  and  which  have  survived  elimination. 
The  choice  of  in  (0,  £*/2)  assures  that  the  procedure  terminates  with  probability  one. 
The  procedure  guarantees  that  the  PCS(6*)  is  at  least  P*.  The  question  of  an  optimal 
choice  of  Si  is  open. 

g-  SELECTION  FROM  BERNOULLI  P.QTOATIQNS:  SS .APPROACH 

As  in  Section  4,  T|, . . .  ,ir*  are  Bernoulli  populations  with  success  probabilities  pi, . . ., 
Pk,  respectively.  Gupta  and  Sobel  [29]  proposed  and  studied  a  single-stage  procedure 
based  on  n  independent  observations  from  each  population.  Let  A*,  denote  the  number  of 
successes  from  Wj,  i  =  1, . . . ,  k.  The  Gupta-Sobcl  procedure  Rbss-.gs  is:  Select  n ,■  if  and 
only  if  Xi  >  max  Xj  —  d,  where  d  —  d(k\  n,  P*)  is  the  smallest  positive  integer  for  which 
the  P* -requirement  is  satisfied. 

Sequential  procedures  are  important  in  practice  when  the  cost  of  sampling  is  high  or 
when  the  observations  are  scarce  so  that  it  is  difficult  to  have  the  sample  size  needed  by  a 
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fixed  sample  size  procedure  in  order  to  achieve  the  desired  level  of  the  PCS.  In  the  Bernoulli 
model,  they  have  the  added  importance  of  ethical  considerations  when  the  experiment 
concerns  comparisons  among  drugs;  one  would  want  a  drug  with  a  small  success  rate  0, 
to  be  identified  soon.  Since  a  subset  selection  rule  also  serves  as  a  screening  procedure 
before  selecting  one  of  the  drugs  as  the  best,  it  makes  sense  to  eliminate  poor  drugs  rather 
quickly  so  that  more  observations  can  be  used  for  the  remaining  ones. 

Recently,  Sanchez  [44]  considered  a  class  of  sequential  procedures  which  take  no  more 
than  n  (common  sample  size  in  Rbss-.gs)  observations  from  each  population  and  result 
in  the  identical  terminal  decision  as  does  Rbss-.gs ■  All  the  procedures  in  this  class  share 
the  same  stopping  rule  S*  and  terminal  decision  rule  T *  (to  be  defined  later).  An  ontima.1 
procedure  in  this  class  is  defined  to  be  the  one  which  minimizes  the  expected  value  of  AT, 
the  total  number  of  observations  taken  until  termination.  In  order  to  determine  an  optimal 
procedure,  we  should  consider  procedures  that  take  observations  one-at-a-time.  However, 
this  turns  out  to  be  a  difficult  task  (see  Sanchez  [44]).  In  this  context,  Sanchez  [44] 
investigated  a  procedure  which  uses  a  modification  of  the  so-called  least-failures  sampling 
rule  of  the  Bechhofer-Kulkarni  procedure  Rbiz-.bk  described  in  Section  4.  Although  this 
procedure  is  not  optimal,  it  seems  to  perform  well  enough  to  be  of  practical  interest. 
Sanchez  has  considered  asymptotic  [44]  ns  well  ns  small  sample  [45]  performance  of  this 
procedure,  the  latter  based  on  simulation. 

We  now  complete  our  discussion  by  formally  describing  the  modified  least-failures  pro¬ 
cedure  of  Sanchez  [44].  Let  n  and  d  be  the  common  sample  size  and  the  constant  of  the 
Gupta-Sobel  procedure  Rbss-.gs •  Observations  are  taken  one-at-a-time.  Let  xlm,  yim, 
and  n;m  denote  the  number  of  successes,  number  of  failures,  and  the  total  number  of 
observations,  respectively,  from  tt*  through  stage  m.  Let  Ss  and  Se  denote  the  subsets  of 
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selected  populations  and  of  excluded  populations,  resj>ectively,  into  which  the  populations 
are  assigned  possibly  at  each  stage  according  to  the  following  rule: 

Assign  7r (  to  Ss  if  a*»m  +  nunyjm  >  n  —  d; 

Assign  TTi  to  Se  if  Vim  +  max  Xjm  >  n  +  d  +  1;  (8) 

No  assignment  is  made  otherwise. 


Sanchez  Procedure.  UjiSS.S-  Least-failures  sampling  is  employed  until  for  some  jr,-, 

n,m  =  n  and  y,m  =  max  y,m  at  which  time  this  tt,  is  assigned  to  Ss-  From  this  stage 
1  <j<k 

on,  additional  observations  are  taken  from  irj  (j  ^  t  )  until  the  first  stage  when  ir,  can  be 
assigned  to  Ss  or  Se  according  to  (8).  Sampling  is  stopped  when  no  population  remains 
to  be  assigned.  The  terminal  decision  is:  Select  all  the  populations  in  5s- 


7.  SELECTION  FROM  EXPONENTIAL  FAMILY 
In  this  section,  we  discuss  some  recent  results  of  Gupta  and  Miescke  [26]  and  Liang  [37] 
for  selection  from  k  populations  belonging  to  a  one-parameter  exponential  family.  Liang’s 
approach  is  classical  with  the  goal  of  CS{6 *),  same  as  that  of  the  Gupta-Liang  procedure 
Rnss-.GL  described  in  Section  5.  Gupta  and  Miescke  [26]  adopted  a  decision-theoretic 
approach  to  sequential  selection.  Their  treatment  includes  multi-stage  selection.  They 
have  obtained  results  for  selection  of  subsets  of  random  as  well  as  fixed  sizes. 

7.1  Liang  Procedure.  Refss:£-  Let  iru...,irk  be  k  populations  where  ir<  has  density 
f(x\8i),  where 


/(x|0)  =  c(0)exp(0x)h(x),  x  real  , 


and  0  €  0,  an  interval  on  the  real  line.  For  specified  6*  >  0,  any  population  7r<  is 
defined  to  be  good  if  0*  >  fyj  —  6*.  Liang  [37]  considered  the  goal  of  selecting  a  subset 
which  contains  the  best  population  and  excludes  any  that  is  not  good  (same  goal  as  that 
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of  Rnss-.ol  in  Section  5).  His  sequential  procedure  with  elimination  is  based  on  certain 
conditional  likelihood  functions  and  it  achieves  the  P* -requirement  for  CS(6*).  The  details 
are  omitted  here. 

7.2  Guota-Miescke  Decision  Theoretic  Approach.  Consider  the  one-parameter  exponen¬ 
tial  family  T  given  by 

T  =  {c(tf)exp(^ar)ft(x),  x  6  R}ee& 

where  0  C  R  is  an  interval.  We  consider  the  class  Vj  of  permutation  invariant  sequen¬ 
tial  procedures  with  or  without  elimination,  employing  vector-at-a-time  sampling.  Let 
Xu  ,  Xtf, . . .  be  a  sequence  of  observations  available  to  the  experimenter  from  x ,  (with 

associated  parameter  8,).  At  stage  m  (m  =  1,2,...),  let  nm  observations  be  taken 

Nm  m 

from  the  eligible  populations.  Let  IV tm  =  S  A'j,-,  where  Nm  =  E  n,-,  denote  the 

>=i  >=i 

sufficient  statistic  for  9,,  based  on  all  observations  from  x,  through  stage  m,  and  let 

Wm=(Wlm . Wkm),  m  =  1,2,.... 

For  8  =  (6X , . . . ,  9k )  €  ft  =  0*,  Lm(Sl\  t . . t,n  ,  fm+i )  denotes  the  loss  incurred  when 

the  procediures  stops  at  stage  m  with  a  record  {fj  ,  tm+i },  where  tj,  j  =  1, . . . , m, 

denotes  the  subset  of  {xi, . . . ,  7r*}  that  is  eliminated  at  stage  j,  and  tm+J  denotes  the 
subset  finally  selected  at  termination.  Note  that  {tj , . . . ,  tm+i }  is  a  disjoint  decomposition 
of  {ffi, . . . , x*}.  It  is  assumed  that:  (a)  Lm  is  permutation  invariant,  and  (b)  Lm  increases 
if  a  record  is  changed  in  any  way  making  a  better  population  eliminated  before  an  inferior 
one. 

A  natural  terminal  decision,  at  stage  m,  selects  only  those  populations  among  the 
noneliminated  ones  which  yielded  the  largest  values  of  lF,m.  Gupta  and  Miescke  [26]  have 
shown  that  between  any  two  procedures  which  differ  only  in  their  terminal  decisions,  the 
procedure  that  uses  a  natural  rule  for  terminal  decision  has  a  smaller  risk. 
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It  is  reasonable  to  speculate  that,  within  stages  where  a  procedure  with  elimination 
does  not  stop,  natural  subset  selections  are  optimal  as  in  the  case  of  terminal  decisions. 
However,  this  has  been  proved  by  Gupta  and  Miescke  [26]  only  in  the  case  of  multi-stage 
procedures  with  the  sizes  of  the  subsets  selected  at  each  stage  fixed,  under  the  assump¬ 
tion  that  T  is  strongly  unimodal  [i.e.  exponential  density  is  logconcave].  For  additional 
comments,  see  Miescke  [38]. 
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8.  CONCLUDING  REMARKS 


As  pointed  out  earlier,  we  have  not  attempted  to  provide  any  sort  of  comprehensive 
survey  of  sequential  selection  procedures.  We  have  discussed  only  a  few  of  the  selection 
procedures  which  are  dealt  with  in  the  books  mentioned  in  Section  1.  These  few  procedures 
are  included  to  make  the  discussion  of  recent  results  contextually  clear.  There  are  other 
problems  of  current  interest  which  are  not  included  here.  For  example,  there  is  some 
interest  in  multinomial  selection  problems  with  truncation  and  curtailed  sampling.  There 
are  several  papers  relating  to  multi-stage  procedures;  especially,  two-stage  procedures. 
These  are  not  included  here.  Also,  we  have  not  discussed  sequential  procedures  for  selecting 
populations  better  than  a  standard  or  control. 
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